Distribution-free, Risk-controlling Prediction Sets

While improving prediction accuracy has been the focus of machine learning in recent years, this alone does not suffice for reliable decision-making. Deploying systems consequential settings also requires calibrating and communicating uncertainty predictions. To convey instance-wise tasks, we show how to generate set-valued predictions from a black-box predictor that controls expected loss on future test points at user-specified level. Our approach provides explicit finite-sample guarantees any dataset by using holdout set calibrate size sets. This framework enables simple, distribution-free, rigorous error control many demonstrate it five large-scale problems: (1) classification problems where some mistakes are more costly than others; (2) multi-label classification, each observation multiple associated labels; (3) labels have hierarchical structure; (4) image segmentation, wish predict pixels containing an object interest; (5) protein structure prediction. Last, discuss extensions quantification ranking, metric learning, distributionally robust learning.